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A commentary on 

Modeling causal conditional reasoning 
data using SDT: caveats and new insights 

hy Trippas, D., Verde, M. E, Hundley, S. /., 
Roser, M. E., McNair, N. A., and Evans, J. 
S. B. T. (2014). Eront. Psychol. 5:217. doi: 
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Signal Detection Theory (SDT; Wickens, 
2002) is a prominent measurement model 
that characterizes observed classification 
responses in terms of discriminability and 
response bias. In recent years, SDT has 
been increasingly applied within the psy- 
chology of reasoning (RoteUo and Heit, 
2009; Dube et al, 2010; Heit and RoteUo, 
2010, 2014; Trippas et al, 2013). SDT 
assumes that different stimulus types (e.g., 
valid and invalid syllogisms) are associated 
with different (presumably Gaussian) evi- 
dence or argument-strength distributions. 
Responses (e.g., "Valid" and "Invalid") are 
produced by comparing the argument- 
strength of each syllogism with a set of 
established response criteria (Figure lA). 
The response profile associated to each 
stimulus type can be represented as 
a Receiver Operating Charateristics 
(ROC) function by plotting perfor- 
mance pairs (i.e., hits and false-alarms) 
along different response criteria, which 
Gaussian SDT predicts to be curvilinear 
(Figure IB). 

Trippas et al. (2014; henceforth 
THVRME) applied SDT to causal- 
conditional reasoning and make two 
points: (1) that SDT provides an 



informative characterization of data from 
a reasoning experiment with two orthog- 
onal factors such as believabUity and 
argument validity; (2) that an inspection 
of the shape of causal-conditional ROCs 
provides insights on the suitability of nor- 
mative theories with the consequence to 
consider affirmation and denial problems 
separately. 

The goal of this comment is to make 
two counterarguments: First, to point 
out that the SDT model is often unable 
to provide an informative characteriza- 
tion of data in designs as discussed by 
THVRME as it fails to unambiguously 
separate argument strength and response 
bias. THVRME's conclusion that "believ- 
ability had no effect on accuracy [. . .] but 
seemed to affect response bias" (p. 4) solely 
hinge on arbitrary assumptions. Second, 
that THVRME's reliance on ROC shape 
to justify a separation between affirmation 
and denial problems is unnecessary and 
misguided. 

1. SEPARATING ARGUMENT 

STRENGTH AND RESPONSE BIAS 

Assume a toy SDT model with four (equal- 
variance) evidence distributions, corre- 
sponding to the four types of syllogisms 
resulting from the Validity (V = Valid/I = 
Invalid) x Believability (B = Believable/U 
= Unbelievable) factorial design. Now, 
let the means of the distributions be 
given by the main effects of Validity and 
Believability as well as their interaction, 
using a 0/1 factor coding. This factorial 
design produces the table in Figure IC. 



The possibility of specifying different 
response criteria for the two levels of the 
Believability factor leads to an uniden- 
tifiable SDT model in which differences 
between means trade-off with differences 
between response criteria (Wickens and 
Hirshman, 2000; Klauer and Kellen, 201 1). 
For example, the ROCs in Figure ID can 
be equally accounted for by a differ- 
ence in the distributions (Figure IE) or 
by a response-criteria shift (Figure IF). 
Because THVRME and others fix IB to 0 
a priori, they enforce a response-criteria 
shift interpretation of the ROCs. This 
ambiguity in the characterization of the 
data compromises the attempt to relate its 
parameters with different accounts on e.g., 
the belief-bias effect. THVRME briefly 
mention this (see their Footnote 2) but 
do not address its implications. The IB 
= 0 restriction implies that effects of 
believability on argument strength can 
only be detected if the interaction term 
is non-zero as the main-effect term of 
believability is effectively censored. This 
means that a pure criteria-shift account 
can be enforced as long as no severe vio- 
lations of additivity (i.e., an interaction) 
are observed. In other words, only when 
VB differs from VU (while assuming IB 
= 0) can the proposed pure criteria-shift 
model be rejected. To make matter worse, 
the criteria-shift account is implausible to 
begin with given that it runs counter to 
empirical work showing that individuals 
do not tend to change their response cri- 
teria on a trial-by-trial basis (e.g., Morrell 
et al, 2002). 
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FIGURE 1 I (A) A graphical representation of the SDT model for a syllogistic 
reasoning task. (B) ROC curve representing the cumulative probabilities for 
hypothetical pairs of hits and false-alarms ("valid" responses to valid and invalid 
syllogisms, respectively) based on the four response categories depicted in 
(A). (C) Factorial design of Believability x Validity representing the means of the 
SDT evidence distributions. (D) ROCs for believable and unbelievable 



syllogisms. (E) Distribution shift account of ROCs in which the distributions for 
believable syllogisms (solid lines) are shifted to the right. (F) Response-criteria 
shift account of ROCs in which the response criteria for believable syllogisms 
(solid lines) are shifted to the left. Note that for ease in the illustration the 
response proportions implied by the SDT accounts of panels (E,F) do not 
exactly correspond to the response proportions depicted in panel (D). 
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2. DATA AGGREGATION CONFOUNDS 
IN CAUSAL-CONDITIONAL 
REASONING 

THVRME's reliance on ROC shape to 
justify the separation between the affir- 
mation and denial problems is unnec- 
essary and misguided: It is unnecessary 
because the acceptance rates (A) already 
show the pattern Amp > Ada and Aac < 
Amt\ indicating that performance is 
"above chance" for affirmation problems 
but "below chance" for denial problems 
(see Singmann and Klauer, 2011, for simi- 
lar results). This contrasting pattern in the 
acceptance rates alone indicates that aggre- 
gating affirmation and denial problems is 
an unwise option. Note that the criticisms 
associated to acceptance rates (e.g., Klauer 
et al., 2000; Dube et al, 2010; Heit and 
Rotello, 2014) do not hold here as they 
are exclusively concerned with the inter- 
pretation of response patterns of the form 
AvB > Avu> AiB > Aiu. 

THVRME's use of eyeball and 
regression-based evaluations of ROC 
shape is misguided because it overlooks 
the more subtle (but still pernicious) 
distortions from item heterogeneity 
(Rouder and Lu, 2005), but also because 
it faOs to characterize SDT's actual abil- 
ity to fit their own data. As it turns 
out, SDT fits the linear aggregate ROCs 
better (VB/IB: G^(3) = 7.95, p = 0.05; 
VU/IU: 0^(3) = 10.63, p = 0.01) than 
the curvilinear ROCs from acceptance and 
denial problems (smallest G^(3) = 13.51, 
p < 0.01). The sufferable fit of the aggre- 
gate data is not surprising given Gaussian 
SDT's ability to account for near-linear 
ROCs when performance is low^. 

3. CONCLUSION 

THVRME attempt to demonstrate the 
value of SDT modeling in research on 
causal-conditional reasoning. However, 



'MP, Modus Ponens; MT, Modus ToUens; AC, 
Afirmation of the Consequent; DA, Denial of the 
Antecendent. 

^ Note that a non-parametric characterization of 
ROCs is possible (Kornbrot, 2006). 



the main motivation for employing 
SDT is to characterize differences in 
argument-strength and response bias 
across conditions. As we have shown, 
the approach of THVRME is unable to 
accomplish this in an unambiguous fash- 
ion. Furthermore, THVRME's detection 
of differences between affirmation and 
denial problems hinges on an evaluation 
of ROC shape that is not only unneces- 
sary (as acceptance rates are sufficient) but 
also fails to relate ROCs with SDT predic- 
tions in a principled way. SDT has a long 
and successful history in psychological 
research, and will likely provide impor- 
tant insights in the reasoning domain; 
however, from the current standpoint, we 
fail to see the exact contribution of the 
SDT modeling advocated by THVRME 
and others (e.g., Dube et al, 2010; Trippas 
et al., 2013; Heit and Rotello, 2014) to 
research on human reasoning. 
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